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STUDY  OF  METHODS  FOR  EVALUATION  OF 
THE  pert/cost  MANAGEMENT  SYSTEM 

ABSTRACT 

The  Department  of  Defense,  in  June  of  1962,  promulgated  PERT/COST 
as  a  new  general  purpose  management  system  for  use  on  major  military 
system  acquisition  programs.  Ir  .ial  implementation  and  testing  of  PERT/ 
COST  are  being  accomplished  by  the  Air  Force  on  the  F-111  (TFX)  weapon 
system  at  the  Aeronautical  Systems  Division  (ASD)  by  a  special  Air  Force 
Systems  Command  (AFSC)  implementation  team  under  the  supervision  of  the 
PERT/COST  subgroup  of  the  AFSC  PERT  Control  Board.  Secondary  applica¬ 
tions  of  pert/cost  are  being  made  at  the  Ballistic  Systems  Division  (BSD) 
on  the  Mobile  Mid-Range  Ballistic  Missile  (MMRBM)  program  and  at  the 
Space  Systems  Division  (SSD)  on  the  Titan  HI  program. 

Mitre  has  investigated  the  question  of  how  to  evaluate  the  design  of  the 
pert/ cost  management  system.  Four  different  ^proaches  have  been 
considered.  This  document  presents  the  results  of  such  effort. 

The  general  conclusion  is  that  there  is  no  single,  simple  straightforward 
way  of  deriving  value  judgments  as  to  the  PERT/ COST  system  design,  or 
probably  any  other  general  purpose  management  system  for  that  matter. 
Because  of  the  unavailability  of  comparable  cases  and  the  lack  of  significant 
quantities  of  cases  for  statistical  techniques,  no  scientifically  recognized 
techniques,  which  exclude  judgment  on  the  part  of  the  observer,  appear 
possible.  Furthermore,  due  to  the  interrelationships  between  a  management 
system  and  the  quality  of  its  implementation  operation  (including  the  csqpability 
of  the  managers  who  use  it),  assessment  of  the  value  of  the  management 
system  alone  presents  serious  difficulties  of  both  a  theoretical  and  practical 
nature. 

Subjective  evaluation  by  use  of  carefully  prepared  questionnaires  appears 
to  be  the  only  feasible  approach  at  this  time.  Additional  effort  to  develop 
techniques  with  an  objective  content  is  recommended.  An  evolutionary 
management  system  development  program  is  strongly  urged. 
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SECTION  I 

INTRODUCTION 

pert/cost  is  the  name  of  a  newly  devised  management  system,  planned 
as  an  improvement  over  the  basic  PERT/TIME  technique.  The  PERT/TIME 
technique  is  a  management  tool  currently  in  use  principally  for  program  planning, 
scheduling  and  status  icvL  .  The  essential  ne-v  characteristic  provided  by 
pert/cost  is  its  integration  of  explicit  program  cost  planning  and  control  with 
the  pert/time  program  planning  and  control  technique.  *  There  have  been 
small-scale  experiments  of  techniques  similar  to  PERT/COST  by  the  Air  Force, 
the  Navy,  and  a  number  of  defense  contractors  over  the  past  two  or  three  years. 

The  Department  of  Defense  (DOD)  and  the  National  Aeronautics  and  Space 
Administration  (NASA)  have  recently  agreed  upon  a  common  general  design 
approach  to  PERT/COST.  The  Air  Force  has  selected  the  F-111  (TFX)  weapon 
system  program  for  pilot  testing  of  this  approved  PERT/COSi  design  approach 
on  a  full-scale  weapon  system  program.  It  has  also  selected  the  TItal  III  and 
the  Mobile  Mid-Range  Ballistic  Missile  (MMRBM)  programs  to  serve  as  secondary 
programs  for  additional  Air  Force  experimentation  with  PERT/COST. 

The  MITRE  Corporation  and  the  Electronic  Systems  Division  (ESD)  of  the 
Air  Force  Systems  Command  (AFSC)  were  initially  requested  to  perform  an 
independent  evaluation  of  the  approved  PERT/COST  management  system  design, 
as  Implemented  on  the  TFX  weapon  system. 


*In  this  memorandum,  the  term  PERT/TIME  will  be  used  to  mean  the  planning, 
scheduling,  and  program  status  assessment  tool,  without  a  coat  dimension. 
pert/cost  will  be  used  to  denote  a  PERT/TIME  technique  integrated  with  a 
cost  planning  and  control  technique.  PERT  will  be  used  to  refer  to  the  technique 
generically,  without  implication  that  either  PERT/TIME  or  PTRT/COST  in  meant. 
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This  document  describes  various  approaches  which  MITRE  considered  in 
an  attempt  to  propose  a  practical  method  of  accomplishing  such  evaluation  on  an 
objective  basis.  Meanwhile,  in  the  absence  of  a  manageable  technique  for 
objective  evaluation,  the  DOD  is  conducting  its  own  evaluation  of  PERT/COST  on 
a  subjective  basis  by  use  of  a  questionnaire  to  all  military  agencies  attempting 
its  use. 
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SECTION  II 

GENERAL  HISTORICAL  BACKGROUND 
PERT/TIME  EXPERIENCE 

The  original  or  "classical"  PERT/TIME  technique  was  developed  initially 
for  the  Special  Projects  Office  of  the  U.S.  Navy' s  Bureau  of  Weapons,  for  use 
on  the  Fleet  Ballistic  Missile  (F'BM)  program.  The  .Navy  approach  involved,  as 
the  author  understands  it,  the  use  of  separate  networks  on  a  contractor-by¬ 
contractor  basis.  Data  are  gathered  and  processed  on  that  basis,  and  manually 
integrated  by  personnel  at  the  Special  Projects  Office.  It  is  generally  understood 
that  PERT/TIME  has  been  an  unqualified  success  on  the  FBM  program. 

Following  the  Navy's  lead,  the  Air  Force  rapidly  adopted  the  PERT  TIME 
technique,  but  applied  it  on  an  nv^r-all  system  basis  (as  opposed  to  the  contractor- 
by-contractor  approach  of  the  Navy).  The  Aeronautical  Systems  Division  (ASD) 
of  AFSC  prepared  its  own  PERT/TIME  computer  program,  ait  improvement 
over  the  Navy  program.  This  Air  Force  program  is  now  known  as  PERT  1. 

The  Ballistic  Systems  Division  (BSD)  experimented  with  PERT  II.  a  PERI'  TIME 
variant  and  computer  program  especially  tailored  to  the  special  requirements  of 
missile  programs.  BSD  and  the  Space  Systems  Division  (SSD)  used  the  over-all 
stems  approach,  but  the  latter  employed  a  variant  of  PERT  TIME,  known  as 
TOPS,  developed  by  the  Aerospace  Corp»>ration.  ESD  and  MITRE  first  used  the 
Navy  system  and  program,  converting  to  PERT  I  when  that  became  available 
in  early  1962. 

Table  I  gives  some  indication  of  the  magnitude  of  the  current  (1963)  use 
of  PERT  as  a  military  systems  management  tool  on  a  system-wide  basis  in  the 
Air  Force. 
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TABLE  I 


Application  of  PERT  in  a  Military  Systems  Management 


- -  ■  -  , —  _ 

ASD 

BSD 

ESD 

SSD 

TOTAL 

Number  of  current 
System  Programs 
Using  PERT 

6 

3 

10 

4 

23 

Number  of  Current 
System  Programs 
Which  Do  Not  Use 
PERT  and  Ne  ver 
Attempted  Its  Use* 

20 

3 

10 

16 

i 

49 

Rough  Estimate  of 
Procurement  Value 
of  System  Programs 

U  sing  PERT 
(in  billions) 

$6 

$13 

$1 

_ 

$1 

$21 

♦Generally,  these  programs  were  initiated  before  the 
PERT  technique  was  available  in  the  Air  Force,  and 
phaseover  to  PERT  was  not  deemed  feasible. 

PERT  is  also  used  foi  nonsystems  projects,  such  as  GFAE  procurement  and 
advanced  planning  and  research.  Such  use  is  beyond  the  scope  of  this  memorandum. 


In  the  Air  Force,  PERT/TIME  did  not  initially  meet  with  unqualified 
success.  Serious  difficulties  were  encountered  on  at  least  the  following  systems: 


AFSC  Division  Program 

ASD  Dynasoar  (PERT  I) 

BSD  Minuteman  (PERT  II) 

465L  (PERT  I) 
SAMOS  (TOPS) 


ESD 

Othei 
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However,  the  apparent  successful  applications  of  PERT/ TIME  seem  to  outnumber 
the  apparent  unsuccessful  applications.  And  the  general  opinion  is  that  PERT/ 
TIME  has  proven  itself  as  an  Air  Force  systems  management  technique.  * 

NAVY  pert/cost  DEVELOPMENTS 

Li  1961,  the  Naw  sponsored  a  PERT/ COST  research  and  development 
effort  by  the  Management  Systems  Corporation  (MSC),  The  effort  involved  a 
survey  of  existing  approaches  to  contractor  program  cost  controls,  a  preliminary 
pert/cost  system  design,  feasibility  tests  to  evaluate  the  preliminary  design, 
and  a  final  PERT/ COST  system  design  document,  incorporating  the  measures 
learned  from  such  feasibility  experiments.  MSC  completed  this  program  in 
April  1962,  releasing  for  review,  at  that  time,  a  preliminary  draft  of  a  docu¬ 
ment  entitled  "The  PERT/ COST  System  Design.  " 

AIR  FORCE  pert/cost  DEVELOPMENTS 

Prior  to  December  1961,  Air  Force  attention  in  the  PERT  field  had  been 
concentrated  principally  upon  making  PERT/tIME  work  effectively.  However, 
experimentation  in  adding  explicit  resource  data  to  PERT  had  been  undertaken 
jointly  by  some  contractors  and  System  Program  Office  (SPO)  directors  at  ASD. 
Techniques  similar  to  PERT/ COST  for  nonsystems  were  also  being  considered 
at  ASD.  At  BSD,  PERT  II  was  being  planned  in  such  a  way  that  it  could 
accommodate  PERT/ COST  when  that  system  was  developed.  Some  Air  Force 
contractors  were  independently  looking  into  the  question. 


’“It  may  be  worth  noting  that  the  author  is  not  aware  of  any  carefully  planned  and 
executed  independent  evaluation  of  PERT/tIME.  It  may  also  be  worth  noting 
that  a  failure  analysis  study  of  the  unsuccessRil  PERT  applications  might  yield 
considerable  di  idends. 

'“’“These  tests  were  conducted  on  portions  of  the  FBM  program  at  the  Lockheed 
Mission  Division,  Sunnyvale,  California,  and  at  the  General  Electric  Ordnance 
Division,  Pittsfield,  Massachusetts. 
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However,  it  is  prot^ably  fair  to  say  that,  up  to  December  1961,  the  PERT 
efforts  at  the  AFSC  Divisions  were  directed  principally  toward  getting  PERT/ 

TIME  operational  on  a  number  of  different  programs  simultaneously.  Attention 
to  PERT  at  AFSC  Headquarters  was  directed  mainly  toward  reducing  the 
differences  in  sq>proach  between  the  several  Divisions,  so  that  there  would  be  a 
single,  uniform  approach  to  PERT/TIME  in  the  Air  Force. 

During  the  week  of  December  4  to  9,  1961,  towever,  AFSC  sponsored  a 
PERT/ COST  conference  at  BSD  Herdquarters.  Conferees  included  representatives 
from  the  AFSC  Headquarters,  the  four  AFSC  system  development  divisions,  the 
Navy,  the  Army,  NASA,  The  MITRE  Corporation,  the  Aerospace  Corporation  and 
the  RAND  Corporation.  This  group  received  briefings  from  1 2  industry  and  manage¬ 
ment  consultant  organizations  on  the  nature  of  their  approaches  to  PERT/ COST 
and  the  status  of  their  efforts.  The  general  conclusion  of  the  conference  was 
that  it  was  time  to  undertake  a  concerted  PERT/ COST  development  effort,  leading 
toward  large-scale  testing  on  a  total  weapon  system  basis. 

Throughout  the  early  months  of  1962,  therefore,  the  Air  Force  proceeded 
with  planning  and  organizational  preparation  to  develop  an  Air  Force  PERT/ 

COST  system.  A  detailed  AFSC  PERT  Management  and  Development  Plan  was 
issued  by  AFSC  Headquarters  in  April. 

DOD/NASA  PERT/COST  DEVELOPMENTS 

Upon  Issuance  of  the  Navy' s  PERT/ COST  system  design  document  in 
April  1962  for  advance  review,  the  separate  Air  Force  and  Navy  PERT/ COST 
design  efforts  were  coalesced.  A  PERT  coordinating  committee  had  been 
previously  established  at  DOD  level  to  provide  coordination  between  the  services 
on  PERT  and  to  furnish  a  point  of  DOD  contact  with  other  government  agencies, 
such  as  NASA,  the  Atomic  Energy  Commission  (AEC),  and  the  Federal  Aviation 
Administration  (FAA).  As  a  result  of  deliberations  at  this  level,  the  Navy/MSC 
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pert/cost  system  design  was  approved,  with  modifications,  and  released 
publicly  as  the  **DOD  and  NASA  Guide,  PERT/COST  System  Design.  ”  dated  June 
1962.  This  document  provides  the  basic  design  of  the  system  which  is  being 
implemented  on  the  F-111  weapon  system  program. 
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SECTION  m 

AIR  FORCE  IMPLEMENTATION  AND  TEST  OF  PERT/COST 
DOD  INSTRUCTIONS 

By  memorandum  dated  1  June  1962,  the  Secretary  of  Defense  officially 
endorsed  the  DOD/nASA  PERT/ COST  System  Design  Guide  for  adoption  by  all 
the  military  services  effective  1  July  1962.  Each  of  the  military  services  was 
subsequently  instructed  to  implement  and  test  PERT/ COST,  on  a  priority  basis, 
on  at  least  one  major  program  in  the  research  and  development  stage. 

For  this  purpose ,  each  service  was  to  establish  a  PERT/ COST  implementation 
team.  The  DOD  further  stated  that  additional  experimentation  and  development 
of  pert/cost  would  not  be  permitted  without  prior  approval.  While  each 
service  was  expected  to  develop  its  own  internal  procedures  for  analyzing  and 
using  the  PERT/ COST  management  summary  reports,  all  such  procedures  were 
to  be  reviewed  the  Office  of  the  Assistant  Secretary  of  Defense  (Installations 
and  Logistics)  to  assure  uniformity. 

AIR  FORCE  SELECTION  OF  THE  F-111  PROGRAM  FOR  TESTING  OF  PERT/COST* 

AFSC,  acting  as  the  responsible  USAF  PERT  control  agency,  appointed 
ASD  as  the  key  division  for  implementing  PERT/COST,  and  selected  the  F-111 
(TFX)  program  as  the  system  program  for  the  first  full-scale  PERT/ COST 
testing.  This  PERT/ COST  effort  on  the  F-111  program  is  to  be  carried  out 
with  hi^  priority,  but  in  such  a  manner  that  it  does  not  provide  major  interference 
with  the  weapon  system  program.  Insofar  as  possible,  therefore,  PERT/COST 
development  and  test  activities  are  to  be  performed  apart  from  the  weapon 

*The  Navy  has  selected  the  Typhon  System  and  certain  FBM  subsystems  for  its 
initial  system  tests  of  PERT/COST.  The  Army  has  selected  the  Mauler  program 
for  pert/ COST  testing. 
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system  program.  The  MMRBM  program  is  presently  designated  as  the  follow-on 
or  second  PERT/ COST  test  bed.  A  third  program  authorized  to  experimentally 
use  pert/cost  is  Titan  III. 

ROLES  OF  PARTICIPATING  ORGANIZATIONS 

The  AFSC  PERT  Control  Board  (PCB)  is  the  official  AFSC  organization 
with  over-all  responsibility  for  tlie  development  of  PERT/ COST  and  its  applica¬ 
tion  and  testing  on  Air  Force  system  programs.  The  PCB  is  responsible  for 
review  and  approval  of  proposed  changes  to,  or  deviations  from,  the  DOD/nASA 
pert/cost  System  Design  Guide  and  the  approved  or  planned  AFSC  PERT 
configurations ,  including  contractor  and  military  service  input  and  output  data- 
reporting  formats.  To  assist  it  in  this  activity,  the  PCB  has  established  a 
pert/ COST  subgroup  to  monitor  all  authorized  PERT/COST  efforts. 

A  special  AFSC  PERT/ COST  implementation  team  has  been  formed  to 
adapt  pert/cost  to  the  F-111  program,  implement  it,  and  assist  in  its  initial 
operation.  The  chief  of  the  implementation  team  is  responsible  for  the  manage¬ 
ment  of  this  effort  and  for  the  detailed  s^plication  of  PERT/COST  procedures 
and  techniques  to  the  weapon  system  program.  The  PEPT/cOST  implementation 
team  has  four  major  subdivisions: 

(a)  Design  and  Development 

(b)  Implementation 

(c)  Organization  and  Manning,  and 

(dO  Integration  and  Analysis. 

Specific  tasks  SoSi^ed  to  the  Design  and  Development  group  are  to  be  accomplished 
by  joint  participation  of  personnel  from  the  PERT  staff  groups  of  ASD,  BSC,  ESD, 
and  SSD  under  the  administration  and  control  of  the  chief  of  the  AFSC  implemen¬ 
tation  team.  Insofar  as  design  details  affect  the  F-111  program,  they  are 
subject  to  the  approval  of  the  F-111  SPO  Director. 


10 


TM-3430 


The  Commander,  ASD,  is  responsible  for  the  conduct  of  the  PERT/COST 
pilot  test  on  the  F-111  program,  including  development  of  procedures,  supervision 
and  control  of  the  AFSC  PERT/ COST  implementation  team,  and  deliniatlon  of 
responsibilities  between  the  Implementation  team  and  the  TFX  system  program 
director. 

The  Commanders,  ESD,  BSD,  and  SSD,  are  to  provide  manpower  and  other 
support  as  agreed  upon  between  representatives  of  the  AFSC  PCB  and  such  divisions. 

The  Commander,  BSD,  assisted  by  personnel  of  the  Aerospace  Corporation 
is  also  to  proxdde  the  implementation  team  with  BSD  representatives  who  will 
not  only  assist  in  the  F-111  PERT/COST  system  application,  but  will  also 
coordinate  and  agree  upon  the  details  of  PERT/ COST  as  it  will  be  applied  to  the 
MMRBM  program  (and  subsequent  BSD  programs).  The  BSD  representatives 
will  provide  the  nucleus  for  a  later  BSD  PERT/ COST  implementation  team. 

The  Commander,  ASD,  is  authorized  to  contract  for  outside  assistance, 
subject  to  the  limitations  and  requirements  of  AFSC  Memorandum,  dated  29  May 
1962,  entitled:  "Use  of  Consultant  Firms  to  Support  Management  Programs.  " 

The  Management  Systems  Corporation  has  been  employed  to  act  in  an  advisory 
capacity  to  the  AFSC  implementation  team  at  ASD. 

PREUMINARY  GUIDELINES  FOR  PERT/COST  EVALUATION 

The  AFSC  PERT  Mani«ement  and  Development  Plan  of  April  1962  (as 
amended)  tentatively  suggests  that  PERT/COST  performance  be  measured,  in 
general,  by  the  capability  of  the  system  to  meet  its  objectives  and,  more 
specifically,  by  certain  particular  criteria  such  as  timeliness  and  regularity 
of  reports,  accuracy  of  data,  etc.  At  the  time  this  Plan  was  issued,  however, 
it  was  well  understood  that  the  method  of  evaluating  PERT/ COST  had  yet  to  be 
worked  out.  This  memorandum,  therefore,  reflects  the  first  comprehensive 
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attempt  to  develop  a  way  or  ways  to  evaluate  PERT/COST  on  an  objective  basis. 
As  such,  it  has  been  written  with  the  material  in  the  AFSC  PERT  Management 
and  Development  Plan  in  mind,  but  not  in  any  way  constrained  by  the  plan. 
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SECTION  IV 

TWO  BASIC  TYPES  OF  EVALUATION 

MANAGEMENT  SYSTEM  UFE  CYCLE  CONCEPTS 

A  management  information  system  (which  is  what  the  PERT/COST  system 
is),  or  any  data  system,  may  be  generally  considered  to  have  a  system  life 
cycle  of  a  type  analogous  to  a  command  information  system  life  cycle,  in  terms 
of  a  conceptual  phase,  an  implementation  (acquisition)  phase,  and  an  operational 
phase. 

In  a  conceptual  phase,  one' s  attention  is  focused  on  activities  such  as  the 
following:  recognition  of  a  need  for  improvement  over  the  current  mode  of 
management  operations,  including  a  management  analysis;  definition  of  the 
functional  requirements  deemed  necessary  or  desirable  to  Improve  the  situation 
to  acceptable  limits;  investigation  of  currently  known  alternative  management 
system  design  approaches  (including  the  current  mode  of  management  operations 
as  one  alternative)  which  will  fulfill  the  functional  requirements,  and  selection 
of  a  preferred  approach;  preparation  of  an  over-all  system  design  concept,  or 
selection  of  a  preferred  system  design  concept  from  among  possible  alternatives; 
and.  finally,  preparation  of  preliminary  system  designs.  Pilot  testing  of  a 
system  prototype  in  a  small  and  controlled  part  of  the  management  environment 
is  probably  the  most  advanced  step  that  mi|^t  be  ascribed  to  the  conceptual  phase. 

The  system  implementation  (^aae  Involves  such  matters  as:  the  writing 
of  detailed  procedures;  establishment  of  data  flow  content,  frequency  and  format, 
writing  of  any  necessary  computer  programs,  acquisition  of  all  necessary  data- 
processing  and  communication  equipment;  training  of  peraonnei  who  must 
provide  data  inputs  to  the  system,  and  indoctrination  of  persons  who  will  use 


13 


TM-3430 


outputs  of  the  system  for  management  decision-making  or  other  action;  and 
integration  of  the  system  with  the  other  management  systems  or  techniques 
alongside  of  which  it  is  to  operate.  It  includes  provision  of  the  initial  operational 
inputs  to  the  system  and  analysis  of  initial  system  operational  outputs  to  assure 
that  the  management  system  is  operating  in  the  manner  intended. 

A  system  operational  phase  involves  use  of  the  system  for  management 
decision-making  and  other  action.  As  the  name  implies,  this  phase  also  includes 
steady-state  operations  for  an  indefinite  period. 

It  is  not  the  purpose  of  this  report  to  explore  in  detail  all  of  the  possible 
steps  or  the  sequence  of  steps  involved  in  management  systems  development 
(such  matters  are,  within  limits,  reasonably  debatable  in  today' s  state-of-the- 
art).  *  Rather,  the  life  cycle  of  a  management  system  is  compared  to  that  of 
other  military  systems  to  point  up  the  fact  that  there  are  two  fundamentally 
different  types  of  system  testing  and  evaluation  in  a  management  system  life 
cycle,  just  as  there  are  in  other  types  of  systems,  namely; 

(a)  "system  design  evaluation,"  which  evaluates  the  adequacy  of 
the  design  of  the  system,  and 

(b)  ’^ystem  operational  evaluation, "  which  tests  whether  the 
system  as  Implemented  is,  in  fact,  performing  as  it  was 
designed  to  perform. 

SYSTEM  DESIGN  EVALUATION 

The  first  type  of  system  evaluation,  "system  design  evaluation"  (or  "system 
design  verification"),  should  be  performed  in  all  phases  of  a  system  life  cycle, 
though  with  different  techniques  in  each  phase.  The  purpose  of  such  continued 
evaluation  through  the  life  cycle  Is  to  assure  sound  (hopefully,  optimal)  system 

•See,  for  example.  AFR  300-2,  AFR  300-3,  AFM  171-9.  AFR  375-1.2.3.4. 
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design.  As  such,  it  involves  continual  analysis  of  the  operational  requirements, 
the  operational  environment,  and  the  proposed  system  design,  as  the  design 
progresses  from  approach  to  functional  specification,  to  concept,  to  preliminary 
design,  to  final  design.  At  each  level  of  detail,  system  design  evaluation  is 
performed  to  evaluate  design  alternatives  and  trade-offs. 

Specifically,  in  the  case  of  a  particular  proposed  PERT/ COST  design,  a 
system  design  evaluation  seeks  to  provide  answers  to  the  following  types  of 
questions: 

I.  Does  the  proposed  PERT/ COST  system  design  meet  the  needs 

of  management? 

(a)  What  are  the  management  requirements  to  be  met? 

(b)  Is  the  design  (at  each  level  of  detailing)  conceptually  sound? 

(c)  What  areas  of  the  system  design  warrant  the  most  attention? 

(d)  Are  there  other  design  concepts  (including  existing  techniques) 
which  are  superior  to  the  proposed  design  (again  at  each  level 
of  abstraction)  ? 

(e)  What  are  the  most  likely  causes  of  system  failure,  and 
what  are  the  consequences  of  failure? 

II.  Will  the  proposed  PERT/COST  system  design  be  compatible 

with  its  proposed  operational  environment? 

(a)  What  is  the  proposed  operational  environment? 

(b)  Is  the  design  conceptually  sound  for  operation  in  such 
environment? 

(c)  Is  there  sufficient  flexibility  in  either  the  environment  or 
the  proposed  system,  or  both,  so  that  they  can  be  modified 
for  compatibility? 

(d)  What  are  the  consequences  of  identified  incompatibilities 
with  the  proposed  system  environment? 


15 


TM-3430 


In  the  conceptual  and  implementation  phases,  the  tools  of  design  evaluation 
(studies,  experimental  simulations,  and  tests)  are  aimed  at  increasingly 
comprehensive  and  accurate  understanding  of  the  needs  to  be  served  by  the 
system,  the  environment,  and  the  design  approach,  (concept  and  details)  as  they 
are  developed.  Before  a  management  system  is  operational,  the  most  compre¬ 
hensive  of  such  tools  is  probably  the  full-scale  pilot  test  of  a  management  system 
prototype.  In  the  operational  phase,  one  can  use  the  system  as  implemented  for 
testing  purposes.  This  provides  a  feedback  to  design  from  real-world  operations. 

Design  evaluation,  at  the  total  system  level  as  well  as  at  the  functional 
and  technical  lower  levels,  is  a  continuous  search  for  better  definition  of  require¬ 
ments,  vaJ.idation  of  proposed  requirements,  and  the  search  for  and  evaluation 
of  alternative  design  approaches.  The  results  of  design  evaluation  take  the  form 
of  guidance  to  persons  responsible  for  system  design.  Properly  employed,  the 
main  thrust  of  design  evaluation  will  be,  as  previously  noted,  to  promote 
preferred  (hopefully  optimal)  system  design. 

This  report  is  addressed,  principally,  to  the  question  of  how  to  perform 
design  evaluation. 

SYSTEM  OPERATIONAL  EVALUATION 

The  second  type  of  system  evaluation  may  be  termed  "system  operational 
evaluation.  "  This  term  refers  to  the  process  of  ascertaining  whether  or  not  a 
system,  which  has  been  designed,  developed,  Installed,  and  brought  to  opera¬ 
tional  status,  does,  in  fact,  operate  in  the  manner  for  which  the  system  was 
designed.  This  type  of  evaluation  does  not  investigate  whether  the  system  design 
is  optimum,  or  even  sound,  but,  rather,  whether  the  stated  system  design 
objectives  have,  in  fact,  been  achieved. 
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In  the  specific  case  of  PERT/ COST ,  an  operational  evaluation  program 
would  be  conducted  to  provide  answers  to  the  following  types  of  questions: 

I.  Does  the  management  system,  as  installed  on  the  TFX  weapon 

system,  meet  the  approved  PERT/ COST  system  design  objectives? 

(a)  What  are  the  system  design  objectives  and  limits, 
functional  and  technical,  if  any?  Are  they  being  met? 

(b)  Are  the  accuracy  and  frequency  of  the  data  within  specified 
limits? 

(c)  Is  the  system  as  reliable  as  the  design  calls  for? 

(d)  Do  the  people,  hardware,  software,  and  operating 
procedures,  separately  and  collectively,  function  as 
they  should? 

(e)  If  deficiencies  are  noted,  can  they  be  corrected? 

It  would  appear  that  the  most  iqppropriate  methods  of  performing  this 
type  of  evaluation  are  field  surveys  and  controlled  tests.  Field  surveys  consist 
of  observing  the  system  in  operation  and  Interrogating  personnel  who  rely  upon 
the  system  or  who  play  an  integral  part  in  various  aspects  of  the  system' s 
operation.  Field  testing  involves  such  steps  as  observing  the  effects  of  feeding 
controlled  information  into  the  system;  introducing  operational  deviations  at 
various  points  to  test  system  sensitivity;  attemptidg  to  "penetrate"  the  system 
(i,  e. ,  deliberately  injecting  a  misleading  rosy  or  bleak  picture);  attempting 
to  "saturate"  it  (i.  e. ,  deliberately  burying  management  under  too  much  data); 
or  attempting  to  "disconcert"  the  system  (e.  g. ,  introducing  program  changes 
more  r{q)idly  than  they  can  be  handled). 

The  results  of  such  tests  can  serve  a  double  purpose.  First,  they  serve 
to  acquaint  all  concerned  with  the  practical  limits  of  reliable  system  operation. 
They  either  confirm  that  the  iq}proved  system  design  requirements  have  been 
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met  in  full  or  that  some  of  them  have  not  been  achieved.  The  consequences  of 
not  meeting  requirements  are  demonstrated. 

Second,  the  results  of  this  type  of  testing  can  be  a  valuable  input  for  further 
system  design  evaluation.  Design  objectives  may  be  met,  but  management' s 
real  needs  may  not  be  attained;  in  such  a  case,  there  is  probably  a  deficiency 
in  the  original  design  requirements.  Conversely,  a  design  objective  may  be 
missed,  but  the  operational  consequences  may  be  significant;  in  such  a  case 
there  was  probably  an  overstatement  in  the  original  design  requirements.  In 
either  case,  a  reconsideration  of  design  concept  may  be  in  order. 
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SECTION  V 

FACTORS  AFFECTING  PERT/COST  EVALUATION 
NEED  FOR  A  BASIC  PERT/COST  EVALUATION 

Many  aspects  can  be  considered  in  evaluating  a  management  system  such 
as  pert/cost.  The  question  is:  what  factors  should  be  evaluated?  Before 
answering  this  question,  a  review  of  some  of  the  alternate  possibilities  for  a 
focus  for  evaluation  is  in  order. 

In  its  most  simple  form,  PERT/ COST  is  a  data  system.  On  the  basis  of 
certain  data  inputs,  it  creates  other  data  outputs.  One  could  evaluate  PERT/ 
COST  as  a  data-processing  and  transmission  system  without  critical  examination 
of  the  quality  of  the  data  inputs  or  the  value  of  the  outputs  (see  Fig.  1). 


Mechanics  of  System  Operation 
and  Data  Manipulation 

-> 

Program  Planning 

Program  Authorization 

Program  Control 

— 

Fig.  1.  Program  Model 

A  slightly  broader  i^proach  to  x  iriRT/ COST  evaluation  would  involve 
separate  consideration  of  the  quality  of  the  outputs  to  management.  The  quality 
of  the  outputs  is  a  function  of  the  internal  characteristics  of  the  PERT/COST 
system  and  of  the  quality  of  the  data  inputs.  The  quality  of  the  data  ii^)ut8 
would,  therefore,  be  included  in  this  type  of  approach.  It  may  be  useful  to  term 
the  inputs  and  the  program  model  as  "Management  Investment"  and  the  outputs 
as  "Management  Returns"  (see  Fig.  2).  The  following  diagram  may  assist  in 
illustrating  this  focus  for  evaluation. 
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MANAGEMENT  INVESTMENT  MANAGEMENT  RETURNS 

INPUTS  ] 
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czmzi  [ 

I — II — I 

/ 

Fig.  2.  Relationship  of  Inputs  versus  Outputs 

Broadening  the  approach  still  further,  one  can  add  consideration  of  the 
cost  of  management  investment  in  PERT/ COST  and  the  benefits  of  the  returns 
furnished  by  PERT/COST.  The  former  involves  the  theoretically  easy  tasks  of 
identifying  and  summing  all  costs  reasoniUsly  attributable  to  making  the  PERT/ 
COST  system  work.  The  latter  involves  serious  difficulties.  PERT/  COST  does 
not  itself  manage  a  program.  It  simply  furnishes  information  upon  which,  one 
hopes,  more  timely  and  better  quality  management  decisions  can  be  made. 
Between  PERT/ COST  and  its  effect  upon  a  military  program  is  management, 
and  management  will  make  decisions  and  take  action  on  all  information  at  its 
command.  Casual  relationships  between  PERT/ COST  outputs  and  their  impact 
upon  the  military  program  may  not  (but,  in  some  cases,  may)  be  identifiable 
(see  Fig.  3). 

A  further  broadening  of  the  evaluation  base  for  PERT/ COST  would 
include  the  impact  of  the  system  on  the  SPO  and  prime  contractor  management 
teams  caused  by 

(a)  the  activities  required  of  each  of  them  in  order  to  make  PERT/ 

COST  operate,  and 

(b)  the  availability  of  the  information  from  PERT/ COST  in  the 
places  and  at  the  times  called  for  1^  the  system. 

This  approach  is  illustrated  in  Fig.  4. 
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Fig.  3.  Effect  of  PERT/COST  on  o  MiHtory  Progiom 


*Includes  one-time  costa  for  PERT/COST  system  implementation  on  the  particular  military  system,  con¬ 
tinuing  costs  for  system  operation  throughout  the  life  of  the  system,  and  perhaps  a  pro-rata  allocation  of 
pert/ cost  R&D  and  computer  programming  costs. 
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It  would  appear  that  at  least  one  final  broadening  of  the  focus  for  evaluation 
is  possible.  PERT/ COST  will  probably  have  some  effect  on  other  military  and 
industrial  management  levels  and  groups;  it  will  probably  also  have  an  effect  on 
various  other  information  reports  not  directly  relevant  to  military  programs. 

In  short,  the  presence  of  PERT/COST  will  affect  the  military  management 
environment  just  as,  conversely,  the  environment  affects  the  system  (see  Fig.  5). 

From  the  foregoing  discussion,  it  seems  apparent  that  there  are  many 
criteria  for  evaluating  PERT/COST.  The  following  categories  are  suggested  as 
focal  points: 

PRIMARY:  The  system  Inputs,  program  model,  and  outputs. 

SECONDARY:  The  program  management  decisions  and  actions— the 
impact  on  the  program  at  both  the  SPO  and  higher 
levels,  and  the  cost  of  the  management  system. 

IGNORED:  All  incidental  effects— the  Impact  on  management 

groups  at  the  SPO,  contractor  and  other  manage¬ 
ment  levels. 

In  the  first  category,  attention  is  directed  to  those  aspects  of  program 
information  acquisition,  structuring,  and  presentation  where  PERT/COST 
involves  use  of  different  (and  presumably  improved)  techniques  over  those  that 
would  otherwise  by  employed.  Evaluation  in  depth  is  recommended. 

In  the  second  category,  less  detailed  evaluation  is  recommended  because 
decisions  and  actions  by  management,  and  their  impact  on  the  program,  involve 
use  of  Information  other  than  PERT/COST.  Also,  the  presence  of  management 
judgment  must  be  taken  into  account.  Factors  extraneous  to  the  function  of 
PERT/COST  necessarily  m  ter.  Whatever  the  focus,  however,  criteria  must 
be  established  for  evaluating  and  measuring  PERT/ COST  against  some  standard 
applicable  to  such  criteria. 
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The  cost  of  pert/ COST  is  considered  to  be  of  secondary  importance  for 
two  reasons.  First,  much  of  the  cost  of  PERT/ COST  would  have  been  incurred 
even  in  the  absence  of  PERT/ COST  for  activities  such  as  developing  program 
plans,  a  work  breakdown  structure,  an  account  code  structure,  periodic 
assessment  of  status,  and  so  on.  While  PERT/ COST  requires  that  many  of 
these  activities  be  performed  in  a  somewhat  different  manner,  the  same  general 
type  of  activity  would  still  have  to  be  accomplished  without  PERT/ COST.  ♦  Con¬ 
sequently,  the  cost  of  pert/cost  will  probably  be  quite  difficult,  if  not  impossible, 
to  disentangle  from  the  cost  of  a  non-PERT/COST  approach;  at  the  same  time, 
it  is  not  expected**to  be  significantly  different.  Second,  the  cost  of  PERT/COST 
should  not  be  considered  vdthout  reference  to  the  savings  (if  any)  to  the  program 
expected.  Such  savings  (if  any)  are  difficult  to  identify  because  of  the  effect  of 
management  judgment  and  the  presence  of  non-PERT/ COST  information  in  the 
decision-making  process. 

Any  impacts  caused  PERT/ COST  upon  SPO  and  other  management-level 

organizations  are  incidental  to  or  side  effects  of  its  use.  For  this  reason,  it 
appears  appropriate  to  ignore  them,  whether  their  value  is  positive  or  negative. 
Should  these  impacts  be  of  large  magnitude,  however,  they  may  warrant  further 
attention. 

In  addition  to  choosing  limits  for  the  PERT/ COST  process,  it  is  also 
necessary  to  select  limits  relative  to  the  management  levels  and  the  military 
program  life  cycle  phases  to  be  considered. 


*  This  proposition  assumes  government  contractors  already  possess  estimating 
and  accounting  systems  capable  of  providing  project  control  information  in 
detail. 

**  This  expectation  is  an  intuitive  one  on  the  part  of  the  author.  Some  diversity 
of  intuitive  opinion  may  be  expected  on  this  point. 
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The  DOD/NASA  PERT/cOST  System  Design  Guide  states  that  the  purpose 
of  pert/cost  is  to  improve  the  management  techniques  at  all  levels  of  manage¬ 
ment.  For  the  purpose  of  any  PERT/ COST  evaluation,  it  is  recommended  that 
"all  levels  of  management"  be  considered  to  include  program  management  at 
only  the  SPO  level,  the  AFSC  Division  and  Headquarters  levels,  and  one  manage¬ 
ment  level  beneath  the  SPO  project  level  (e.g. ,  contractor  project  management). 

It  is  recognized  that  DOD,  USAF  Headquarters,  USAF  Logistics  Command,  USAF 
Training  Command,  and  USAF  Using  Commands  and  others,  are  also  levels  of 
military  management  concerned  with  the  planning,  progress,  and  status  of 
military  systems  acquisition,  or  parts  thereof.  Similarly,  there  are  industrial 
management  levels,  above  and  below  those  mentioned  above,  vitally  interested 
in  the  planning,  progress,  and  status  of  a  program,  or  parts  thereof.  However, 
to  keep  the  evaluation  ei^ort  manageable,  it  is  probably  satisfactory  to  continue 
attention  to  the  four  management  levels  mentioned  above. 

The  DOD/NASA  PERT/COST  System  Design  Guide  also  states  that  PERT/ 
COST  is  designed  to  meet  the  needs  of  managers  at  all  steps  in  the  life  of  a 
program.  In  the  acquisition  of  a  major  military  system,  there  are  at  least 
three  major  different  types  of  activities  of  particular  importance: 

(a)  program  planning 

(b)  program  authorizing  and  directing  (selecting  program  participants, 
contracting  with  industry,  negotiating  interagency  charters  with 
supporting  government  agencies,  etc.),  and 

(c)  program  controlling  (including  any  partial  replanning  and  reauthorizing 
necessary  from  time  to  time). 

The  foregoing  steps  fall,  principally,  the  the  Program  Definition  and  Acquisition 
Phases  of  the  life  cycle  of  a  military  system  program.  Accordingly,  for  purposes 
of  any  PERT/ COST  evaluation,  it  is  recommended  that  the  Conceptual  Phase  and 
the  Operational  Phase  of  the  military  system  program  not  be  considered. 
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LACK  OF  AN  OBJECTIVE,  QUANTITATIVE  STANDARD 

One  way  to  evaluate  a  management  system  is  to  ascertain  whether  it  fulfills 
(or  will  fulfill)  some  objective,  applicable  standard. 

In  the  case  of  PERT/ COST  (and  perhaps  other  management  systems,  for 
that  matter),  there  is  no  preestablished  objective,  quantitative  standard. 

Probably  the  closest  thing  to  a  standard  is  the  statement  of  PERT/ COST  design 
objectives  in  the  DOD/nASA  PERT/COST  System  Design  Guide  namely: 


Complex  research  and  development  projects  can  be  managed 
effectively  if  project  managers  have  the  means  to  plan  and 
control  the  schedules  and  costs  of  the  work  required  to  achieve 
their  technical  performance  objectives.  The  serious  schedule 
slippages  and  cost  overruns  that  have  been  experienced  on 
many  weapon  and  space  programs  indicate  that  managers  at 
all  levels  need  improved  techniques  at  all  stages  in  a  project  to: 


—  define  the  work  to  be  performed; 

~  develop  more  realistic  schedule  and 
cost  estimates  based  on  the  resources 
planned  to  perform  the  work; 


—  determine  where  resources  should  be 
applied  to  best  achieve  the  time,  cost, 
and  technical  performance  objectives; 


identify  those  areas  developing  potential 
delays  or  cost  overruns  in  time  to  permit 
corrective  action. 


For  example,  managers  at  each  level  must  be  able  to  determine: 


—  whether  the  current  estimated  time  and 
cost  for  completing  the  entire  project 
are  realistic; 

—  whether  the  project  is  meeting  the  com¬ 
mitted  schedule  rnd  cost  estimate  and, 
if  not,  the  extent  of  any  difference; 
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—  whether  requirements  for  manpower  and 
other  resources  have  been  planned 
realistically  to  minimize  premium  costs 
and  idle  time; 

—  how  manpower  and  other  resources  can  be 
shifted  to  expedite  critical  activities; 

—  how  manpower  and  other  resources  made  available 
by  changes  in  the  project  tasks  can  best  be  utilized. 

The  pert/cost  system,  an  extension  of  the  basic  PERT/ 

TIME  system,  has  been  developed  to  meet  these  planning 
and  control  needs  of  each  level  of  management. 

At  present,  therefore,  the  PERT/ COST  design  objectives  for  the  F-111 
program  are  relative.  They  will  be  "met"  (literally  at  least)  by  any  improvement 
achieved  in  the  above  factors  through  the  use  of  PERT/ COST. 

In  the  absence  of  an  independent  effort  in  investigate  and  determine 
objective,  quanltlative  standards,  it  is  necessary  to  conclude,  at  this  point,  that 
any  evaluation  must  be  accomplished  by  means  which  do  not  require  such  over¬ 
all  standards. 

LACK  OF  A  COMPARABLE  ALTERNATIVE 

Another  way  to  evaluate  a  management  system  is  to  compare  the  results 
achieved  in  two  or  more  comparable  cases,  one  or  more  of  which  uses  PERT/ 
CCM3T  and  one  or  more  of  which  does  not  use  PERT. 

However,  each  military  program  Is  unique;  there  is  no  other  program 
which  is  comparable.  Other  programs  with  other  contractors  and  other  SPOs 
Involve  different  military  systems,  different  technical  and  management  problems, 
different  contract  structures  and  different  management  teams.  One  might 
consider  other  military  programs  on  which  the  prime  contractor  participated  in 
a  major  capacity  In  the  past.  In  the  case  of  the  F-111,  with  General  Dynamics 
(Ft.  Worth)  as  the  prime  airframe  contractor,  it  would  be  the  B-58  weapon 
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system  program.  However,  this  program  preceded  the  F-111  program  by  some 
six  to  eight  years,  had  different  technical  and  management  problems,  a  different 
contract  structure,  and,  in  fact,  a  substantially  different  management  team.  In 
addition,  a  further  difficulty  with  such  a  comparison  is  that  the  B-58  program 
did  not  use  the  basic  PERT/TIME  management  system.  This  would  make  it 
very  difficult  to  separate  any  advantages  of  PERT/ COST  from  those  which 
might  more  properly  be  attributed  to  PERT/TIME. 

INFEASIBILITY  OF  A  STATISTICAL  APPROACH 

Theoretically,  another  way  to  evaluate  PERT/ COST  on  an  objective  basis 
would  be  to  utilize  an  approach  in  which  use  or  non-use  of  PERT/ COST  is 
assigned  randomly  to  a  number  of  programs.  It  would  then  be  possible  to  use 
statistical  methods  to  determine  whether  there  is  a  significant  relationship 
between  use  of  PERT/ COST  and  accomplishment  of  program  objectives.  The 
number  of  programs  which  would  be  needed  to  obtain  significant  results  depends 
upon  the  similarity  of  the  program.  This  approach  suffers  from  two  major 
difficulties: 

(a)  a  technique  for  measurement  of  program  success  or  failure  and 
the  time  lag  Involved  in  the  process,  and 

(b)  the  necessity  for  random  assignment  of  controls  in  the  manage¬ 
ment  of  major  national  defense  programs. 

This  approach  does  not  appear  feasible  as  a  practical  matter. 
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SECTION  VI 

APPROACHES  TO  PERT/COST  EVALUATION 
OBJECTIVE  EVALUATION  BY  MANAGEMENT  TASK 

One  approach  considered  in  depth  for  an  objective  evaluation  of  PERT/ 

COST  was  based  upon  the  proposition  that,  basically,  all  of  the  management 
activities  required  to  carry  out  PERT/ COST  are,  in  one  wiqr  or  another,  present 
in  every  other  thorough-going  approach  to  military  program  management.  That 
is,  pert/cost  does  not  involve  any  essentially  new  management  function  but, 
rather,  provides  a  new  technique  for  fulfilling  them. 

The  concept  was  that  the  smallest  basic  pieces  of  PERT/ COST  can  be 
individually  tested  and  evaluated  first.  (These  pieces  are  referred  to  in  the 
DOD/NASA  pert/cost  System  Design  Guide  as  ’’Management  Tasks.  ”)  Then 
the  pieces  could  be  combined  into  meaningful  management  aggregates,  say,  the 
program  planning  stage,  the  program  authorisation  and  direction  stage,  and  the 
program  control  stage,  for  further  testing  and  evaluation.  Finally,  PERT/(X)6T 
could  be  evaluated  on  an  over-all  system  basis. 

Appendix  I  seta  forth  in  detail  an  approach  to  evaluation  of  PERT/ COST  by 
analysis  of  management  tasks.  It  contains: 

(a)  a  brief  statement  of  the  objective  of  each  management  task 
required  by  PERT/COST; 

<b)  a  statement  of  the  probable  Impact  of  the  use  of  PERT/ COST 
(subject  to  verification  in  the  acUia!  evaluation); 

(c)  typical  questions  om  must  answer  to  evaluate  the  particular 
task  in  question  separately; 

(d)  Possible  criteria  applicable  in  each  case;  and 

(e)  some  pertinent  comnmnts. 

The  advantages  of  this  approach  are  that  the  system  is  broken  into  pieces 
small  enough  to  enable  development  of  more  precise  evaluation  criteria.  At 
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such  level  of  detail,  moreover,  it  may  be  possible  to  make  a  decision  on  objective 
grounds  between  the  management  task  in  PERT/ COST  and  its  equivalent  task 
using  a  pert/time,  standard  cost  management  technique. 

This  approach,  however,  also  presents  several  difficulties.  The  major 
shortcoming  is  that  it  is  directed  toward  the  input  side  of  the  management 
system,  that  is,  a  basic  assumption  is  that  if  each  of  the  necessary  tasks  to 
provide  management  with  needed  information  is  improved,  progran.  management 
will  be  improved.  It  views  the  management  system  through  the  eyes  of  the 
management  information  system  staff,  not  the  managers  whom  the  system  serves. 

The  next  difficulty  with  the  approach  is  that  a  method  of  aggregating  is 
not  readily  apparent.  While  this  ai^roach  eases  the  problem  of  lack  of  compara¬ 
ble  alternatives,  it  does  not  really  resolve  the  problems  mentioned  in  Section  IV. 

OBJECTIVE  EVALUATION  BY  MANAGEMENT  FUNCTION 

The  next  aK>roach  considered  for  an  objective  evaluation  of  PERT/COST 
was  baaed  upon  the  proposition  that  in  order  to  manage  a  program,  there  are 
certain  management  decisions  and  actions  (generally  referred  to  as  functions) 
which  must  be  conducted.  They  involve  identifying  certain  features  of  the  program, 
making  certain  decisions,  structuring  a  program  team,  communicating  certain 
author iaationa,  and  so  forth.  Such  functions  muat  be  performed  whether  or  not 
PERT/COST  is  used. 

Ute  concept  was  that  it  should  be  possible  to  define  all  such  major  manage¬ 
ment  functions.  When  this  is  established,  it  should  be  possible  to  determine 
whether  or  not  the  use  of  PERT/CX)6T  offers  any  improvement  for  a  particular 
hinctlon  of  program  management.  Then  the  pieces  could  be  combined  into 
meanliMful  numagement  aggr^ates.  sigr,  the  same  three  as  used  lit  the  manage¬ 
ment  tmtk  approach,  wltib  some  gmieralisations  about  the  system  as  a  whole. 
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An  advantage  of  this  approach  is  that  the  system  is  being  evaluated  from  the 
viewpoint  of  a  manager  using  the  system.  Appendix  II  sutn  forth  this  approach 
to  evaluation  of  PERT/ COST  on  the  basis  of  management  function. 

There  are  several  difficulties  in  this  approach.  It  does  not  resolve  the 
fundamental  problems  noted  in  Section  IV.  Moreover,  the  several  subfunctions 
noted  in  Appendix  II  are  probably  more  subject  to  debate  than  are  the  nisnage- 
ment  tasks  noted  in  Appendix  I.  Finally,  the  criteria  by  wh^ch  one  will  compare 
pert/ cost  against  a  standard  or  an  alternate  become  more  nebulous  and  less 
quantifiable. 

While  the  management  function  approach  is  probably  theoretically  preferable 
to  the  management  task  approach,  due  to  its  orientation  toward  the  management 
system  user,  it  appears  to  be  much  more  difficult  to  carry  out  as  a  practical 
matter. 

OBJECTIVE  EVALUATION  USING  THE  DOD/NASA  PERT/COST  SYSTEM 
DESIGN  GUIDE 

A  third  approach  to  the  evaluation  of  PERT/ COST  was  considered.  The 
starting  point  of  this  af^roach  was  the  statement  of  PERT/uOST  system  design 
objectives,  quoted  earlier  in  this  Section.  These  objectives  are  listed  on  the 
following  page. 
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dod/nasa 

Objective 

1.  Improved  techniques  to  define  the  work 
to  be  performed 

2.  Improved  techniques  to  develop  more 
realistic  schedule  and  cost  estimates 
based  upon  the  resources  planned  for 
such  work 


3.  Imp’*''  3d  techniques  to  determine  how 
beSi  j  apply  the  resources  to  achieve 
time,  cost  and  technical  objectives  and 
minimize  idle  time 


4.  Improved  techniques  to  determine  how 
best  to  shift  resources  for  expediting 
critical  activities  and  to  utilize  re¬ 
sources  made  available  by  task  changes 

5.  Improved  techniques  to  determine 
whether  the  project  is  meeting  the 
committed  schedule  and  cost  esti¬ 
mate  and,  if  not,  the  extent  of  any 
difference 


Applicable 
Program  Stage 

Planning 

Planning 

Planning 

Control 

Control 
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Stated  somewhat  more  simply,  the 
Planning  Stage 

Identification  of  work  to  be 
performed 

Realistic  schedules 

Realistic  cost  estimates 


DOD/NASA  PERT/COST  objectives  are: 
Control  Stage 

Program  progress  and  cost 
status  monitoring 

Comparison  of  status  with 
authorized  plans  —  devia¬ 
tion  anticipation 


Efficient  application  of  Replanning,  reauthorization 

resources  over  time  as  necessary  to  compensate 

for  inadequate  planning, 
changes,  and  deviations 

It  can  be  observed  that  the  program  planning  stage  represents  PERT/ COST 
in  a  static  mode.  Types  of  criteria  that  can  be  applied  to  this  mode  are  shown 
as  follows: 


Criteria  Relating  to  Planning  Realism 

Accuracy 
Inclusiveress 
Precision 
Nonambiguou  sne  s  s 

Dependencies  and  constraints  explicit 
Ground  rules  and  assumptions  explicit 

Criteria  Relating  to  Planning  Usefulness 

Clarity 

Simplicity 

Correlatabillty  of 

Work  to  be  done 

Military  system  design 

Available  resources 

Authorized  resources 

Schedules 

Estimated  cost 

Dependencies  and  constraints 
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Criteria  Relating  to  Management  Environment 

Correlatability  of  plans  to: 

Technical  fields  of  interest 
Air  Force  management  structure 
Contractor  management  structure 

The  program  authorization  stage  similarly  represents  a  static  mode  of 
pert/cost.  But  since  the  DOD/NASA  guide  does  not  include  any  design 
objectives  relating  to  the  authorization  stage,  this  stage  will  be  ignored  for 
present  purposes. 

The  program  control  stage,  on  the  other  hand,  represents  a  dynamic  mode 
of  pert/ cost.  To  the  extent  the  control  stage  involves  replanning,  the  previous 
criteria  listing  is  relevant.  In  addition,  other  types  of  criteria  also  apply  to 
this  djmamlc  mode: 

Criteria  Relating  to  Data  Communication  and  Processing 

Appropriateness  of  data  sources 
Appropriateness  of  data  recipients 
Efficiency  of  communications 
Efficiency  of  data  processing 
Quantity  of  data 
Accuracy  of  data 
Precision  of  data 

Criteria  Relating  to  Data  Usefulness 

Relevance  of  data 

Timeliness  of  data 

Regularity  of  data 

Clarity  of  data  presentation 

Penetrability 

Satur  ability 

Disconcertability 

Criteria  Relating  to  Management  Environment 
Simplicity  of  operatio.i 

Compatibility  with  Air  Force  management  structure 
Compatibility  with  contractor  management  structure 
Compatibility  with  personnel  motivation 
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In  any  evaluation  of  PERT/COST,  one  must  recognize  that  PERT/COST 
serves  three  major  management  functions:  planning,  authorization  and  direction, 
and  control.  These  functions  can  be  considered  separately  —  one  can  use 
pert/cost  for  planning,  but  not  authorization  and  control;  one  can  use  PERT/ 
COST  for  planning  and  authorization,  but  not  control  —  or  as  a  whole.  In 
order  to  perform  an  evaluation  of  PERT/ COST  as  a  whole  within  the  framework 
of  the  DOD/nASA  design  objectives,  it  is  necessary  to  assign  degrees  of  relative 
importance  to  the  several  objectives  noted.  It  would  appear  desirable  to  first 
make  a  gross  allocation  of  weights  between  the  planning  stage  and  the  control 
stage.  On  the  grounds  that  the  former  is  an  indispensible  forerunner  of  the 
latter,  and  that  better  planning  (and  authorization)  will  ease  Uie  problem  of 
program  control,  let  us  iq>ply  a  60:40  weighting.  That  is,  for  PERT/ COST  as 
a  whole,  planning  accounts  tor  60  percent  of  the  value  and  control  for  40  percent. 

Next,  within  planning  function,  let  us  further  assign  weights  to  the  relative 
importance  of  the  four  ennumerated  DOD/NASA  objectives.  On  the  gprounds 
that  identification  of  the  work  to  be  performed  is  the  primary  step  about  which 
the  others  revolve,  let  us  assign  to  it  a  weight  double  that  of  each  of  the  other 
three  (see  Table  2). 


Table  2 

Planning  Stage  Efforts  versus  Objectives 


Objectives 

Bases  (%)  ] 

100 

60 

I  dentification  of  Work  to  Be 
Performed 

40 

24 

Realistic  Schedules 

20 

12 

Realistic  Cost  Estimates 

20 

12 

Efficient  Application  of 
Resources  over  Time 

20 

12 

Totals 

100 

37 


TM-3430 


Table  3 

Planning  Stage  Objective  No.  1: 
Identification  of  Work  to  Be  Performed 


Criteria 

Weight  (%) 

Inclusiveness 

20 

Accuracy 

20 

Explicitness  of  Dependencies  and  Constraints 

20 

Explicitness  of  Ground  Rules  and  Assumptions 

20 

Clarity 

5 

Simplicity 

5 

Correlatability  of  Work  to  Be  Done,  Military 

Itystem  Design,  Available  Resources, 

Authorized  Resources,  Schedules,  Esti¬ 
mated  Cost,  Dependencies  and  Constraints 

10 

Total 

100 

L- 

SUBJECTIVE  EVALUATION 


An  alternative  approach  to  objective  evaluation,  with  its  inherent  difficulties, 
is  subject  evaluation,  which  can  be  used  to  derive  value  judgments  about  PERT/ 
COST.  Such  judgments  may  well  constitute  the  best,  and  indeed  only,  source  of 
informed  opinion  of  the  benefits  and  limitations  of  PERT/ COST  for  some  time 
to  come. 

A  subjective  evaluation  would  be  carried  out  means  of  questionnaires 
to  and  interviews  with  responsible  persons  who  may  be  expected  to  be  informed 
on  the  management  value  of  PERT/ COST  to  them  as  key  personnel  in  the  program 
management  team.* 

*See  "Management  Information  Systems  Evaluation  Methodology," 

C.  C.  Joyce,  Jr. ,  Mitre  W-6221. 
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The  bulk  of  exploratory  effort  Into  ways  of  evaluating  PERT/COST  was 
directed  to  objective  techniques  rather  than  subjective  ones.  Consequently, 

this  document  will  not  discuss  the  benefits  and  limitations  of  the  subjective 
£q>proach,  except  to  indicate  its  existence  as  an  avenue  of  possible  action. 

DESIGN  EVALUATION  CONCLUSION  CATEGORIES 

Since  the  PERT/COST  evaluation  is  actually  planned  and  the  empirical 

data  gathered,  it  should  be  done  in  such  a  manner  so  that: 

poor  results  due  to  inadequate  implementation  can  be 
separated  from  poor  results  due  to  poor  system  design; 

(b)  poor  results  due  to  inadequate  program  management 
judgment  can  be  separated  from  poor  results  due  to 
poor  system  design; 

(c)  Benefits  attributable  to  PERT/ COST  can  be  separated  from 
those  obtainable  from  PERT/TIME  coupled  with  other  cost 
planning,  correlating  a;:d  control  techniques. 

(d)  it  can  be  concluded  that  PERT/ COST  is  valuable  for 
program  planning,  but  not  necessarily  so  for  program 
authorization  and  direction  or  program  control; 

(e)  it  can  be  concluded  that  PERT/ COST  is  valuable  for 
program  planning  and  program  authorization  and 
direction,  but  not  necessarily  so  for  program  control;  and 

(f)  it  is  possible  to  ascertain  that  PERT/^COST  is  particularly 
valuable  for  decisions  and  actions  at  the  SPO  Director 
level,  but  not  necessarily  for  AFSC  Headquarters  or 
AFSC  Division  Headquarters  levels  or  for  contractor 
managements  (or  any  permutation  and  combination  of 

the  lUbQve). 

While  definition  would  be  desirable  in  a  great  number  of  other  sureas  as  well, 
the  areas  cited  are  probably  the  larger,  involved  ones  which  must  be  dealt 
with  in  order  to  say  anything  meaningful  about  the  value  of  PERT/ COST . 
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SECTION  VII 

CONCLUSIONS  AND  RECOMMENDATIONS 

NO  STRAIGHTFORWARD  WAY  TO  EVALUATE  PERT/COST 

The  major  conclusion  of  this  study  is  inescapable:  there  is  no  straight¬ 
forward  way  to  evaluate  PERT/COST.  The  value  of  tlie  system  is  Intimately 
related  to  both  the  quality  of  its  implementation  and  the  capability  and  willing¬ 
ness  of  the  appropriate  managers  to  use  It.  The  consequences  of  using  the 
system  can  be  ramified  indefinitely.  Military  programs  are  not  comparable, 
and  standards  do  not  exist.  While  value  judgments  about  PERT  and  PERT/ 
COST  can  be  made  meaningfully  by  those  experienced  in  their  use,  Uiere  is  no 
available  methodology  or  established  skill  base  capable  of  evaluating  PERT/ 
COST  professionally  on  either  £ui  objective  or  a  subjective  basis. 

SUBJECTIVE  DESIGN  EVALUATION  FOR  IMMEDIATE  PURPOSES 

In  the  absence  of  a  clear-cut  approach  to  an  objective  evaluation  of  PERT/ 
COST,  the  DOD  is  proceeding  with  a  preliminary  PERT/ COST  evaluation  on  a 
subjective  basis  by  means  of  carefully  prepared  questionnaires  to  the  services, 
divisions  and  SPO  Directors  or  their  equivalents.  This  appears  to  be  the 
correct  approach  at  this  time,  since  it  is  feasible,  and  since  no  objective 
alternate  can  be  proposed.  It  should  be  recognized,  however,  that  as  much 
care  and  effort  should  gu  into  preparing  a  subjective  evaluation  as  into  an 
objective  one,  if  the  data  v'^htaiued  are  to  provide  a  sound  basis  for  meaningful 
judgments  about  the  value  of  PERT/ COST. 

The  DOD’ s  current  approach  of  questioning  the  results  on  all  programs 
using  pert/ COST  is  better  than  the  original  concept  of  evaluating  PERT/ COST 
only  on  the  F-111  program.  This  approach  will  help  to  disentangle  the  cause 
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and  effect  relationships  attributable  to  PERT/COST  from  those  attributable  to 
individual  system  iclios3mcra8ie8. 

It  is  doubtful  that  any  program  has  been  using  PERT/COST  long  enough 
to  have  significant  results  from  its  operation.  *  The  tangli  ^  benefits  to  date, 
if  any,  from  PERT/ COST  may  be  expected  to  be  deri*:  1  fr  i  its  statir  mode 
in  program  planning  and  program  authorization  and  direction. 

NEED  FOR  DEVELOPMENT  OF  EVALUATION  TECHNIQUES 

The  evaluation  of  management  systems,  generaUv.  is  a  abject  that 
appears  not  to  have  been  explored  in  depth  as  yet.  The  literature  on  the  subje  H 
is  meagre  and  unrewarding.  Techniques  for  evaluating  vari  ms  other  t^npes  of 
systems,  both  military  and  data  systems,  have  been  developed,  but  their 
possible  adaptation  for  management  systems  has  not  yet  (apparently)  been 
attempted. 

The  need  to  develop  a  methodology  and  skills  for  evaluating  management 
systems  design  covers  not  only  the  after-the-fact  evaluations  of  systems  in 
the  field,  but  also  tools  for  design  verification  ana  validation  which  can  i)e 
employed  to  assist  better  design  while  a  mimagement  system  is  still  in  its 
conceptual  stage.  Two  approaches  would  seem  to  have  great  potential.  One  is 
to  Investigate  the  use  of  system  design  simulation  for  management  systems  - 
possibly  utilizing  the  evolving  technology  associated  with  ESD' s  System  Design 
Laboratory  for  electronic  systems.  The  other  le  to  sponsor  the  catalogiaing 
of  the  various  management  system  designer' s  real  life  design  constraints  — 
dealing  with  such  matters  as  human  factor  design  limitations,  data-handling 
lag  times,  security  provisions,  and  similar  factors.  An  evolving  management 
system  designer' s  handbook  (patterned  somewhat  after  the  various  designer' s 

♦As  of  June  19(^3. 
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handbooks  pioneered  at  ASD)  would  be  of  great  value  in  validating  the  building 
blocks  of  system  design. 

The  methodology  should  encompass  subjective  evaluations  as  well  as 
objective  ones,  because  it  is  probable  that  the  theoretical  and  practical  difficulties 
of  the  objective  approach  will  necessitate  some  mixed  subjective/objective 
aji^roach  to  be  used  indefinitely. 

THE  NEED  FOR  RECOGNITION  OF  AN  EVOLUTIONARY  APPROACH  FOR 
MANAGEMENT  SYffTEMS  DEVELOPMENT 

The  PERT/C06T  system  has  alreac^  passed  thr<H:gh  a  number  of  steps  in 
the  normal  management  system  life  cycle.  The  general  recognition  of  need 
occurred  in  19G0-61.  Management  analyses  and  preliminary  PERT/COST  system 
design  w-ere  accomplished  in  1961-62,  The  general  DOD/NASA  PERT/COST 
System  Design  Guide  appes-red  in  1962,  and  the  specific  Air  Force  manuals  in 
1963.  An  approved  system  concept  and,  in  tact,  design  details,  has  been 
officially  approved  for  implementation  today.  In  short,  only  one  system  alterna¬ 
tive  is  currently*  binder  consideration,  although  there  are  some  variations  in  its 
proposed  applications  to  various  systems. 

It  has  proven  useful  to  plan  the  development  of  some  Air  Force  command 
systems  on  an  evolutionary  basis,  that  is,  a  controlled  m>iltistsge  effort  (see 
Fig.  6a)  instead  of  a  single  one-time -through  life  cycle  (see  Fig^  Tlie 
timing  of  die  stages  in  Fig.  €1^  is  planned  so  that  the  lessons  learned  from  pre¬ 
vious  stages  can  be  made  available  for  design  of  the  system  in  later  ones.  On 
management  systems,  operations  under  an  early  stage  of  model  of  ti^  system 
are  iwt  converted  to  a  later  one  until  the  napabiliiy  of  tJhe  later  stage  or  model  has 
been  adequately  demonstrated.  If  tlie  t^se-over  involves  too  mimh  an  effort  for 
apy  one  particular  program,  it  is  possible  for  that  program  to  use  the  old,  out¬ 
moded  system  through  to  program  completlcm. 

•As  oflune  1963 
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Fig.  6.  Evolutionory  Oev^lopnmnt  venus  Single  Life  Cycle 


The  evolutionary  a{H)roach  appears  to  be  moat  useful  in  situations  where 
the  ohjectlvea  of  the  system  cannot  or  have  not  been  clearly  defined.  It  is  Ideal 
in  cases  where  the  ultimate  capablUty  to  be  requl;>.  j  of  the  system  cannot  be 
for>^seei,  but  where  the  direction  toward  which  Increasing  system  capabilities 
should  be  oriented  Is  predictable.  In  abort,  an  evolutionary  approach  is  s  good 
tachni^ue  for  eootrollli^  the  development  of  a  ayatem  capability  In  an  orderly 
faehion  over  period  of  time. 

PERT/C08T  appears  to  belong  In  the  claas  of  systems  which  benefit  from 
use  of  Uie  cvolul^nary  development  concept.  For  example,  the  system  has 
already  evolved  from  Navy  PERT  to  Air  Force  PERT  I.  PfcRT  D.  and  PERT 
m.  PERT/cmr,  or  pert  IV.  as  U  is  referred  to  in  Fig.  7,  it  not  being 
considered.  It  is  apparent  to  all  who  are  close  to  the  present  effort  that  PERT 
rv  is  not  the  ultimate  in  military  program  management  syatems.  but  only  a 
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stepping-stone  toward  even  better  management  systems  in  the  future.  Hence, 
we  are,  in  fact,  already  participating  in  an  evolutionary  development  type  of 
effort.  This  fact  should  be  recognized  and  used  as  a  cornerstone  of  future  Air 
Force  and  DOD  planning  for  future  management  systems  development. 

Initiative  in  PERT  matters  was  originally  exercised  in  a  number  of 
quarters  (Navy,  ASD  and  BSD).  What  has  actually  occurred  up  to  June  1963, 
together  with  a  forecast  of  a  centrally  coordim  ted  future  development  effort, 
is  shown  in  Fig.  7. 

Figure  7  also  reflects  some  of  the  overlappiner  and  duplication  of  effort  in 
this  field  which  has  occurred  to  date  because  initiative  in  the  development  of 
management  system  has  been  exercised  at  the  field -operating  division  level. 

If  an  orderly  process  of  management  system  design  improvement  is  to  be 
achieved,  it  is  essential  that  the  process  be  centrally  controlled  and  that  future 
improvements  are  planned  so  as  to  take  advantage  of  the  design  evaluations  is 
prior  stages.  It  would  seem  unnecessary  to  proceed  on  a  "concurrent"  basis 

to  develop  and  implement  proposed  additional  management  system  improvements 
before  earlier  management  system  stages  are  understood  and  evaluated. 


R.  L.  Hamilton 
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